


TER\ 





E-ISSN No : 2454-9916 | Volume: 4 | Issue: 2 | Feb 2018 


TIL: AN INNOVATIVE TOOL FOR THE RECRUITMENT OF 
BACHELOR ENGINEERING STUDENTS IN ITALY 





Maria Giulia Ballatore | Laura Montanaro’ | Anita Tabacco 


‘ Dept. of Mathematical Sciences “G.L. Lagrange’, Politecnico di Torino, Italy. 


* Dept. of Applied Science and Technology, Politecnico di Torino, Italy. 


ABSTRACT 





This paper summarizes the guidelines and the main results of the set-up of an access Test in Laib -TIL performed at Politecnico di Torino, Italy, during the last decade. 
Many reasons forced to define this proprietary tool, such as the introduction of the numerus clausus and the consequent need of an effective and robust evaluation 
process associated to merit, the improvement of the attractiveness at the national and international level, and the reduction of the drop-out rate. This test was able to 
supply several functions, useful both for the university and the potential students. It demonstrated to be a reliable and predictable tool for evaluating the competence 
background needed for a successful career in a technical university. In addition, it is a self-assessment test for the applicants able to contribute to a conscious individual 


choice. 
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INTRODUCTION: 

Literature context: 

In literature students selection has been proposed as a useful tool to rank from a 
pool of applicants, especially when demand exceeds supply (Harman 1994), but 
also to improve the quality of the student population, consequently reducing the 
drop-out rate, as well as the duration of the career (Schaefler 1962); (Selim e Al- 
Zarooni 2009); (Pascoe 1999); (Urpo 1990). 


The need for developing selection mechanisms able to assure equity and fairness 
in higher education was already stressed (Pitman 2016); (Geschwind 2017). A 
selection process must be transparent, simple and equitable and it can be associ- 
ated with merit, 1.e. with talent, skill, intelligence, ability, and effort (Liu 2011), 
one of the main interests of the university being to acquire the most talented stu- 
dents. 


The introduction of an entrance examination is based on the belief that earlier 
school success does not necessarily lead to success in higher education (Ahola e 
Kokko 2001); (Selim e Al-Zarooni 2009); (Graeffe 1989). Therefore today, of 
particular importance, is the predictability of the test, even ifin some cases litera- 
ture supports a better predictor function for medicine (Strupler Leiser e Wolter 
2015); (Mighiaretti, et al. 2017); (Alia e Ali 2010), nurse (Gale, et al. 2016) and 
STEM (Schultz and Austin 1987) than for business administration (Ahola e 
Kokko 2001), and social work students (Lyrén 2008). 


If the student’s point of view is considered, the test should be designed to pro- 
mote a self-assessment exercise, to be flexible and available many times to give 
the student the opportunity to fill the potential gaps (Eitel, Benito e Scheiter 
2017); (Dunlosky e Rawson 2015) and to limit failures induced by emotional situ- 
ations (Daniels e Gierl 2017). In addition, it can aid in limiting the gender gap. In 
fact, although female perform as well as male students in engineering, women’s 
self-perception of their own performance and the confidence in their skills are 
often lower than that of male students (Godwin, et al. 2015). 


It has been highlighted that the high school grades alone cannot give correct indi- 
cation about the career choice (Qian, Chie Bai 2014) and they cannot predict the 
generalist standardized test results (Godwin, et al. 2015) when engineering 
courses are concerned due to the specific characteristics. In fact, in middle and 
high school students, usually, do not have direct engineering experience, while 
they achieve a better perception of other scientific fields, like mathematics, phys- 
ics or chemistry (Marra, et al. 2009); (Seymour e Hewitt 1997); (Williams, 
Engerman e Fleming 2006). 


University background and objectives: 

In the international frame above summarized, Politecnico di Torino (from now 
on referred to as PoliTo, see www.polito.it) developed its own selection strate- 
gies. To better understand the decision taken and the tool adopted, some informa- 
tion concerning the institution as well as its context are given in the following. 


PoliTo, founded in 1859 on the basis of the French Ecole Polytechnique model, is 
an Italian Technical University and it provides training in the macro-areas of 


Engineering and Architecture. 


PoliTo applies the Bologna model, that is the first level of training (Bachelor, BS, 


identified in Italian as Laurea) lasting three years, and a second level (Master of 
Science, MS, Laurea Magistrale) lasting two years, with a possible follow-up ina 
three years’ doctoral path. 


It offers 50 educational programs, 40% of which are BSs, split into the various dis- 
ciplines of Engineering (industrial, ICT, civil-environmental, management) and 
Architecture (more details can be found at (Politecnico di Torino s.d.)). The Ital- 
ian university system is based on credit-accumulation (180 credits for BS and 
120 credits for MS) and each exam is passed if the score ranges between 18 and 
30. At PoliTo to gain access to the second year, students must overcome the 
blocking threshold of 28 credits, that is approximately half of the annual credits 
to be passed. 


The access to the Architecture BS program, which is limited in the number of 
enrolments at a national level, is regulated by a unified test established by the Ital- 
ian Ministry of Education, University and Research (MIUR), on which the single 
University has no way to intervene. 


For the Engineering programs, PoliTo provided an admission process since the 
’90s, at first based on an aptitude, not selective, paper-based test developed at a 
national level by CISIA, a national interuniversity consortium (Consorzio 
Interuniversitario Sistemi Integrati per l'Accesso 2014). In 2007, the first trial of 
an online test run in collaboration with CISIA involved 200 students. The previ- 
ous experience as well as this trial, evidenced several limitations, in particular the 
need of: 


¢ agreater flexibility, that is the possibility to offer several test dates during the 
year and an immediate feedback of the test results; 


¢ an effective assessment of the aptitudes required for successfully attending 
the programs in a technical university thanks to a tailoring of the test contents 
and relative precise reporting of the specific gaps; 


¢ ahigher freedom in managing technical aspects of the informatics platform 
as wellas the income and outcome data. 


To overpass these limits the Moodle platform (Moodle s.d.) was integrated into 
the PoliTo web-portal to provide the design and development of an online test 
starting from the academic year (a.y.) 2008/09: the so-called Test in Laib (TIL). 
The test set-up went on until a.y. 2012/13 in which it turned from aptitude into 
selective, as a consequence of the introduction of the numerus clausus, as further 
detailed. 


Thanks to the many years’ experience as well as to the broad statistical database 
collected, TIL can guarantee transparency, robustness, and effectiveness. The 
test is carried out online by using a proprietary hardware and software, also 
allowing to better respond to the progressive, relevant increase in enrolment 
applications. In fact, several test dates are available starting from February each 
year; the applicant can repeat the test monthly and the last result achieved, 
expressed as a fraction of 100, is used for the ranking. A threshold of 50/100 has 
been set, above which the admission is guaranteed. 


In the a.y. 2017/18 the total number of students enrolled in the BS and MS pro- 
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grams at PoliTo is about 33,000, of which about 4,500 freshmen in the BS Engi- 
neering programs. The attractiveness of international students grew progres- 
sively over the years; now around 5,000 units, about 15% of the total population, 
come from more than 100 foreign countries. 


In fact, in contrast with the national data (Viesti 2016) but also with the situation 
in some other western Countries (Prieto, et al. 2009); (Morice 1990), during the 
last years, PoliTo has experienced a progressive increase in the number of appli- 
cations for admission to Engineering BSs. This trend has been also enhanced by 
the relevant employment rate of its graduates, since PoliTo stands at the first 
place worldly, considering the indicator “graduate employment rate” in QS grad- 
uate employability ranking (QS 2017). This in spite of the high number of stu- 
dents graduated every year: for instance, in 2016 the BS and MS graduates were 
around 3,200 and 3,300, respectively. 


Due to this growing demand, starting from the a.y. 2012/13 a numerus clausus of 
5,000 was set, and it progressively decreased, and it was fixed at 4,500 since the 
a.y. 2015/16. In fact, its value is defined annually on the ground of sustainability 
assessments that refer to available human and infrastructural resources; it also 
affected by stringent national regulatory constraints, in terms of the spending 
review and limited turn-over of the retired teaching staff. In particular, the num- 
ber of professors with permanent contracts at PoliTo is decreasing year by year, 
only compensated by temporary contracts (Table 1). 


Table 1: Professors population at PoliTo year by year 




































# professors with permanent # professors with 
contract temporary contract 
| 2011 | . 839. | - 0 4: 
DON 811 24 
2013 788 25 
2014 764 31 
2015 761 43 
2016 747 | 107 
2017 729 161 

















To perform a reliable and robust selection, the TIL was designed and refined in 
such a way as: 


¢ to guarantee the applicants to be evaluated independently from the high 
school diploma, mostly affected by a relevant inhomogeneity in grades (at a 
national level) and/or in contents (if other countries are concerned); 


¢ to guarantee the applicant's freedom of choice and flexibility in time and 
place since the test is carried out in several Italian regions as well as in vari- 
ous non-European countries. This can improve the attractiveness of national 
and international students since it limits the travel and attendance costs to be 
incurred without any certainty of the results; 


¢ tobereliable in the assessment of the aptitudes and of the effectiveness of the 
previous training, by achieving a significant correlation between test results 
and students' careers; 


¢ to setup a self-assessment tool for each applicant regarding his/her previous 
training path and its soundness for approaching an Engineering BS, associ- 
ated to the possibility of performing an on-going effort to fill any training gap 
for passing the admission test in due time; 


¢ tobeaimed at improving the overall quality of the freshmen and reducing the 
drop-out rate, through a more targeted selection at the entrance; 


e to maintain a very high employability rate, thanks to the assessed quality of 
the graduates which can profit from a careful selection of students' input; 


¢ tocreate the background for a training project aimed at the most gifted stu- 
dents (top 2%) which has proved to be a further stimulus to choose PoliTo as 
a privileged training place. This specific project will be detailed in a further 
contribution. 


The present paper is aimed to describe this evaluation tool and to comment on its 
validity, through objective and statistically based results. A particular attention is 
addressed to the correlation between the test result and the success of the aca- 
demic career, and the consequent reduction of the drop-out rate. Only the Italian 
students are here considered, in order to compare a homogeneous set of people 
who have attended a similar high school training path. 


MATERIALS AND METHODS: 

The data considered in this study refer to all the Italian pre-enrolled as well as 
enrolled students from the a.y. 2010/11 (that is from the entry into force of the so- 
called Gelmini reform, Italian Law 270 ), as detailed in Table 2. 
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Table 2: Number of pre-enrolled and enrolled students year by year 
acting as a source of the database 











Enrolled 





Type of test | a.y. Pre-Enrolled 




















Aptitude fr 4032 
2011/2012 5328 | 3989 
Selective 2012/2013 5232 3677 
2013/2014 5780 4168 
2014/2015 6190 4432 
2015/2016 7591 4518 | 
2016/2017 























For each student, the following data are available: gender, high school diploma 
type, grade, year, and place. When a student enrols, other fields of the database 
are populated: identification number, year of enrolment, family financial status, 
credits and average grade earned at the end of each a.y., career end-date, last a.y. 
of enrolment, and BS degree mark. 


The TIL consists of 42 multiple-choice questions of varying difficulty divided 
into 4 sections related to the following subject areas: mathematics, physics, ver- 
bal comprehension, and logic. The total available time is 1.5 hour. The math sec- 
tion is made of 18 questions to be answered in 40 minutes, the physics part is com- 
posed of 12 questions to be solved in 26 minutes, while the verbal and logical 
comprehension parts globally consist of 12 questions in 24 minutes. Multiple 
answers, precisely 5 for each question, are provided, only one of which is correct. 
The score 1s calculated by assigning | point for each correct answer; in the event 
of an incorrect answer, 0.25 points are subtracted. The ungiven answers are not 
taken into account. The result of the test is automatically calculated and 
expressed as a fraction of 100, and it can be recovered at the end of the test as well 
as anytime on the personal web-page of each candidate. To eventually support 
the students in filling the specific knowledge gaps, some comments and training 
aids (video recording of classes, educational games, online booklets, and test 
training platform) are available together with the numerical results. 


At the end of each exam section, the global and detailed results record are added 
to the PoliTo database. 


RESULT AND DISCUSSION: 

Increasing attractiveness and freshmen quality: 

PoliTo attractiveness increased after the introduction of the numerus clausus, sig- 
nificantly expanding the number of candidates to be selected. The graph in Fig- 
ure | shows that the asymptote in enrolled students imposed by the numerus 
clausus was reached in the last years, while the curve plotting the pre-enrolled stu- 
dents vs. time shows a clearly positive derivative, especially in the last two aca- 
demic years. 
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(solid line: pre-enrolled; dashed line: enrolled) 


Figure 1: Trend of pre-enrolled and enrolled students year by year 





Ina.y. 2010/11, 80% of the applicants were enrolled after the test, whereas in a.y. 
2016/17 this percentage dropped to 50%. 


Even if the number of pre-enrolled students continuously increases, the male- 
female ratio remains almost constant (about 3:1) over time, although the total 
number of pre-enrolled students increases (Figure 2a). The same proportion is 
maintained also in the enrolled student population (Figure 2b). 
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percentage of students 


percentage of students 


(grey: male, black: female) 


Figure 2: Gender population as percentage year by year: 
a) pre-enrolled, b) enrolled students 


These data are in agreement with the gender ratio in other technical universities 
in Europe (CESAER 2015) and in other countries (GE Fund 2002); (Yoder 
2014). 


The number of students coming from PoliTo’s region, Piedmont, located in the 
north-west of Italy, remained almost constant over the years, whereas the fresh- 
men coming from the other Italian regions increased (Figure 3). 
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(grey: Piedmont; black: other Italian regions) 


Figure 3: Number of enrolled students as a function of the region of 
origin 


It is reasonable that the PoliTo’s choice to establish a selective test has inspired 
confidence for the investment of households for off-site study, particularly rele- 
vant in a period of deep economic crisis such as the one object of this survey. 


This is an anomalous trend in the Italian context, where there is a strong tendency 
to study close to the city of origin, being the universities offering engineering pro- 
grams uniformly distributed throughout the country (Vergolinia e Zanini 2015). 


The trend of the average TIL result during time gives evidence of a global 


improvement of the competence background of the students, fostered by the 
numerus clausus and the selective test (Figure 4). 
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(solid line: Piedmont; dashed line: other Italian regions) 


Figure 4: Relation between region of origin and average TIL score 


Starting from the a.y. 2012/13, that is when the numerus clausus was introduced, 
a progressive increase of the average score was recorded, corroborating that a 
selective test acts as a strong stimulus to improve the own competence back- 
ground and consequently the success rate. Moreover, the students from outside 
Piedmont present a higher average quality. This is probably related both to a 
stronger personal motivation, which pushes them to move from the region of ori- 
gin, and to the greater economic investment requested to their own families. 


The introduction of a numerus clausus can also lead to a self-selection process 
that the potential student usually makes on the ground of the diploma grade only 
(the Italian high school diploma is considered passed if the grade is between 60, 
minimum, and 100, maximum score). However, the results of the TIL well high- 
light the lack of an imputable correlation between the Italian diploma grade and 
the competence background for a successful access and path into an Engineering 
BS program (Figure 5). For sake of simplicity, the TIL scores are grouped into 6 
categories: A (TIL score lower than 20), B (TIL score between 20 and 34), C (TIL 
score between 35 and 49), D (TIL score between 50 and 59), E (TIL score 
between 60 and 69), and F (TIL score between 70 and 100). 
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Figure 5: Relation between TIL score and high school diploma grade 
on the total number of enrolled students (datum given in Table 2) 





The high school diploma grade seems to be better related to the most extreme test 
results. On the other hand, in the case of test grades between 20 and 59, the high 
school grade is not relevant on a sound statistical basis, as the rate of success is 
spread out fairly evenly. 


Predictive power of the test: 

The test is strongly aptitude and predictive, as one can easily notice by consider- 
ing some indicators of the success of the carrier of the students enrolled from the 
a.y. 2010/2011 to the a.y. 2014/15 (the latest a.y. useful to collect a complete 
career overview in a three years’ lasting course). 


The first indicator, i.e. the total number of credits earned at the end of each BS 
year, outlines a very good correlation to the TIL result (Figure 6). 
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number of credits earned 
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BS year 


Figure 6: Correlation between the total number of credits earned and 
the TIL score category 


In the meantime, a similar trend is observable correlating the average exam 
scores at the end of each BS year to the TIL result, see Figure 7. 


—— A (<20) 
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BS year 


Figure 7: Correlation between average exam scores and TIL score 
category 


A third indicator is the global carrier duration. As shown in Figure 8a, more than 
60% of the students of the category F graduated in 3 years. On the contrary, 
almost all the students of the category A did not succeed in completing their stud- 
ies within the legal duration. Considering a broader time frame that is up to 4 
years from the first enrolment, the students of the category F are almost all gradu- 
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ated, against only 10% of students of the category A (Figure 8b). 


TIL score 
—<— A (<20) 
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TIL score 
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—o— B (20-34) 
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percentage of students graduated 
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Figure 8: Percentage of students graduated in 3 years (a) and within 4 
years (b) as a function of the TIL score category 





A more detailed analysis can be performed by focusing on the result of each TIL 
section. First of all, the relation between the global career duration and the score 
of each section is discussed, remembering that the weight of math is 18/42, that 
of physics is 12/42, as well as that of logic and verbal comprehension. Again, for 
a sake of simplicity, the scores are grouped into categories: A (lower than 3), B 
(from 3 to 7.99), and C (from 8 to 12); for the mathematics one more category is 
added: D (from 12.01 to 18). 


In, similarly to , the global carrier duration 1s related to the score of each TIL sec- 
tion. 


percentage of students graduated 


percentage of students graduated 
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Figure 9: Percentage of students graduated in 3 years (a,b,c) and within 4 years (a’,b’,c’) as a function of each TIL section and related categories 


(math (a), physics (b), and logic and verbal comprehension (c); mixed line: A; dashed line: B; dotted line: C; solid line: D) 


The mathematics (a) and physics (b) parts have a preponderant weight in the pre- 
dictive role of the test, while the scores of logic and verbal comprehension (c) are 
of lower importance. However, the relative position of the curves for the different 
categories remains the same for each section. As a consequence, the global struc- 
ture of the test results well balanced. 


Minimum test score requirement and first-year blocking threshold: 

Starting from the a.y. 2015/16 a minimum test score of 20/100 must be achieved 
in order to be included in the final ranking. The choice of this score is justified by 
the strong correlation between the TIL results and the student's career (see again 
Figure 6, Figure 7, and Figure 8). In addition, due to the important attractiveness, 
and the numerus clausus, the lowest TIL score to be enrolled was over 30/100 in 
the last few a.ys.. 


On the other hand, as a confirmation of the growing quality of the incoming stu- 
dents, the number of them who pass the blocking threshold (28 credits) at the end 
of the first year is gradually increasing during the time (Figure 10). 
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In addition, a strong correlation can be observed between the overcoming of the 
blocking threshold and the TIL score (Figure 11), once again confirming the 
effectiveness of TIL as well as supporting the introduction of the minimum test 
score. 


average TIL score 


Figure 11: Average TIL score of students who passed (solid line) or 
failed (dashed line) the first-year blocking threshold year by year 
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Drop-out reduction: 

The soundness of the test is also witnessed by a traceable decrease in the drop-out 
rate over the years. The number of active students is continuously growing, in 
spite of the numerus clausus, whereas the number of those who renounce within 
the first year of enrolment is decreasing in percentage (Figure 12). 
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Figure 12 Trend of active (solid line), drop-out within 1“ year (dotted 
line) and false drop-out within 1° month (dashed line) a.y. by a.y. 


On the other hand, it can be observed that some students give up within the first 
month of the a.y., that is by October 31“ (false drop-out in Figure 12). These stu- 
dents cannot be considered in the drop-out rate, since their leave is mostly imput- 
able to a fail in synchronization of the national tests (for instance, for the enrol- 
ment in Medicine and Architecture studies) and consequently to a late transfer to 
another BS course. The most gifted candidates attend more than one the access 
tests and after that, they make their choice. In fact, it was observed that the num- 
ber of these false drop-out increases with the increase of the TIL score. 


In addition, considering also the drop-out rate within two, three, and four years, it 
has been observed that half of them are concentrated within the first year (Figure 
cy, 


percentage of drop-out students 


(mixed line: within 1“ year; dashed line: within 2 years; dotted line: within 
3 years; solid line: within 4 years) 


Figure 13: Distribution of the drop-out population a.y. by a.y. 
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CONCLUSIONS: 
TIL demonstrated to be: 


areliable and robust tool for the students’ selection in view of the enrolment 
inatechnical university; 


predictable as concerns the university career goodness; 
flexible in terms of availability in time and place; 


a self-assessment exercise for the potential students to verify the quality of 
their own competences. 


This tool, together with the introduction of a numerus clausus, the adoption of 
some restrictive policies in terms of minimum test score for the admission in the 
final ranking, and of a blocking threshold for the access to the 2nd year, has pro- 
moted some beneficial consequences. One can mention, an increase of the attrac- 
tiveness and the average quality of the students, and a decrease in the drop-out 
rate. All these positive achievements are a benefit for PoliTo, to fulfill the 
sustainability requirements using a valuable selection tool based on merit. In 
addition, they contribute to a conscious individual choice, limiting the societal 
drawbacks associated with a wrong decision about the university career. 
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